Skip to content

fix(quality): keep the gate verdict if the summary write fails - #237

Merged
tbhb-agent merged 3 commits into
mainfrom
agent/rules-mutation-summary
Nov 14, 2026
Merged

tbhb-agent merged 3 commits into
mainfrom
agent/rules-mutation-summary

Conversation

@tbhb-agent

Copy link
Copy Markdown
Contributor

Why

mutation_gate_test appends its timeout line to $GITHUB_STEP_SUMMARY. After #233, a summary path that failed to write ended the gate before its verdict. Bazel's test sandbox also dropped the variable, so the summary got nothing.

Behavior change

A failed write to the summary now prints mutation_gate: could not write to GITHUB_STEP_SUMMARY on stderr. The gate still prints its timeout line and survivor listing, and it exits with the verdict the report deserves.

mutation_gate_test now inherits GITHUB_STEP_SUMMARY from the environment. The timeout line lands in the job summary when CI sets the variable. The change also drops a comment that quoted one run's timeout figures.

Decisions

  • The rule returns RunEnvironmentInfo(inherited_environment = ...). env_inherit belongs to native test rules, and this rule is custom.
  • CI passes the variable through the rule, so the workflow doesn't change.
  • The failing-report test checks the stderr note, because that report fails the gate with or without the guard.

Risk

Low. The change touches the append and the environment list. Bazel keys the test's cache on the inherited value, so a new summary path reruns the gate test.

Verification

A scratch file as GITHUB_STEP_SUMMARY, one gate test, no --test_env: before the change the file was empty, and afterward it held mutation_gate: 5 timed out of 100 (5.0%), counted as killed.

With the stderr guard removed, both new tests fail. The passing-report test exits 1 instead of 0, and the failing-report test doesn't find the stderr note.

The tests cover a missing directory only, with no read-only file. CI doesn't show whether a gate inside a real job receives the variable.

Opened-by: LLM effort=medium;harness=claude-code:2.1.322;model=anthropic:claude-sonnet-5-5;session=sha-256-64:debb814921f029ad;surface=cli
Generated-by: LLM effort=medium;harness=claude-code:2.1.318;model=anthropic:claude-sonnet-5-5;session=sha-256-64:f1227f050617f845;surface=sdk-cli
Assisted-by: LLM effort=medium;harness=claude-code:2.1.322;model=anthropic:claude-sonnet-5-5;session=sha-256-64:debb814921f029ad;surface=cli

A GITHUB_STEP_SUMMARY that cannot be written no longer ends the gate under set -e before the verdict and survivor listing; it warns on stderr. The gate test rule now inherits GITHUB_STEP_SUMMARY, which Bazel's test sandbox otherwise drops, and the stale timeout figures leave the cap comment.

Generated-by: LLM effort=medium;harness=claude-code:2.1.318;model=anthropic:claude-sonnet-5-5;session=sha-256-64:f1227f050617f845;surface=sdk-cli
Generated-by: LLM effort=medium;harness=claude-code:2.1.318;model=anthropic:claude-sonnet-5-5;session=sha-256-64:f1227f050617f845;surface=sdk-cli
The summary warning's own write to stderr could end the gate before its verdict when stderr was closed. Ignore that failure, require the FAIL line in the failing summary test, say no mutant failed when none was checked, and reword the header.

Generated-by: LLM effort=medium;harness=claude-code:2.1.318;model=anthropic:claude-sonnet-5-5;session=sha-256-64:06dc52093e5d8aba;surface=sdk-cli
@tbhb-agent
tbhb-agent merged commit 53f0f1d into main Nov 14, 2026
104 checks passed
@tbhb-agent
tbhb-agent deleted the agent/rules-mutation-summary branch November 14, 2026 15:51
tbhb-agent added a commit that referenced this pull request Nov 14, 2026
Why
The gate read each report with `grep ... || true`. That treats grep
status 2 (an unreadable file or a broken pattern) like status 1 (no
match), so a failed read looked like a clean report. Under fault
injection, six fixtures that must fail exited 0. Review of
#237 found it, and it predates that PR.

Behavior change
Every verdict read goes through one helper. Status 0 and 1 pass as
before. Any higher status prints the report and the step on stderr and
exits 1. The helper serves the value reads, the mutant and count reads,
the scope read, the unmapped count, and the timeout listing. A failure
in any stage of the unknown-status pipeline fails the gate.

Reports that parse cleanly get the same verdict and exit status as
before. The comment before the final message now separates a diff that
selected nothing from an all-skipped report, which also checks nothing
in full scope.

Decisions
- The timeout listing fails closed too. A listing that can't run would
  hide the timeouts it exists to show.
- The summary-write guard from #237 keeps its form,
  because it protects a file write and the gate verdict comes from the
  report.
- One helper serves every site. A grep added later has one obvious way
  to handle status.
- Tests put a shim ahead of the real tool on `PATH` that returns an
  exact exit status. File permissions are unchanged.

Risk
Low for well-formed reports. A gate on a host whose grep fails for an
unrelated reason now fails where it used to pass, and that's the intent.

Verification
Six status-2 fixtures exited 0 on the previous main, and all exit 1 now.
Thirteen fault tests inject status 2 at the grep sites and at each stage
of the unknown-status pipeline. A shim prints valid output before it
fails, and one test uses status 3.

Putting back `|| true` on a count read, on the pipeline or on the scope
read fails the matching test with `the gate exited 0, not 1`.

The faults come from shims. The tests never read a real unreadable file.

A different build host ran the builder checks, because the usual one was
unreachable.

Generated-by: LLM effort=medium;harness=codex:0.164.0;model=openai:gpt-6.1-sol;session=sha-256-64:44399538b20b63d8;surface=exec
Assisted-by: LLM effort=medium;harness=claude-code:2.1.322;model=anthropic:claude-sonnet-5-5;session=sha-256-64:debb814921f029ad;surface=cli
tbhb-agent added a commit that referenced this pull request Nov 14, 2026
Why
A pull-request mutation lane needs to report surviving mutants without
blocking the merge. The gate fails on any survivor today, so a lane
can't report one. Nightly lanes must keep blocking.

Behavior change
`mutation_gate_test` takes `advisory`, which defaults to False. With it
set, lived and not-covered mutants no longer fail the gate. It lists
each survivor and prints
`mutation_gate: advisory, N survivors (not failing)` on stdout and in
the step summary. Both runner macros, `mutmut_test` and
`gomutants_test`, forward it.

Errored mutants, interrupted runs, bad accounting, malformed reports,
timeouts over the cap, unmapped mutants, an empty full run and a diff
report under full scope still fail.

Decisions
- Advisory covers survivors only. A second advisory case needs its own
  PR.
- The docstring says blocking nightly runs leave `advisory` False.
- The advisory line reuses the guarded summary append from
  #237. A failed summary write gives a stderr warning and
  exit 0.
- The survivor listing goes through the helper from #239.
  A failed read fails the gate.

Risk
Low while `advisory` is False. A lane that turns it on sees survivors as
a passing check.

Verification
Each non-survivor failure still exits 1 in advisory mode, and nine
proofs cover them. The same survivor report exits 1 without advisory.

Forcing advisory on fails the proof that survivors fail without it.
Ignoring advisory fails the survivor test. Letting advisory wave through
errored mutants fails the errored proof.

The build ran on a different build host because the usual one was
unreachable. Remote CI, other operating systems and runner integration
with advisory on are unverified.

Generated-by: LLM effort=medium;harness=codex:0.167.2;model=openai:gpt-6.1-sol;session=sha-256-64:d27b03000ac671ee;surface=exec
Assisted-by: LLM effort=medium;harness=claude-code:2.1.322;model=anthropic:claude-sonnet-5-5;session=sha-256-64:debb814921f029ad;surface=cli
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant